LAGRANGIAN SUBBUNDLES OF SYMPLECTIC BUNDLES 

OVER A CURVE 



INSONG CHOE AND GEORGE H. HITCHING 

Abstract. A symplectic bundle over an algebraic curve has a natural invariant SLag 
determined by the maximal degree of its Lagrangian subbundles. This can be viewed 
' as a generalization of the classical Segre invariants of a vector bundle. We give a sharp 

upper bound on SLag which is analogous to the Hirschowitz bound on the classical Segre 
O ■ invariants. Furthermore, we study the stratifications induced by SLag on moduli spaces of 

symplectic bundles, and get a full picture for the case of rank four. 



Q 



, 1. Introduction 



Let X be a smooth projective curve of genus g > 2 over C. Let G be a connected 
reductive algebraic group over C, and P C G a parabolic subgroup. Given a G-bundle V 
over X, we have an associated G/P-bundle vr: V/P — ?> X. For a section a of vr, consider 
the normal bundle N^^ over cr(X) = X in V/P. We define 

> 

:=min{degA^^}, 

where a runs through all sections of V/P. 
. It is easy to show that s(V; P) > —oo; see Holla-Narasimhan O Lemma 2.1]. A section 

■ a is called a minimal section of V if degA'o- = ,s{V;P). According to Ramanathan 

Definition 2.13], y is a semistable G-bundle if and only if s{V;P) > for every maximal 
^ ■ parabolic subgroup P of G. In general, it can be said that the invariant s{V; P) measures 

^ ' the degree of stability of V with respect to P. The invariant s(— , P) is a lower semicontin- 

uous function on any parameter space of G-bundles over X, so defines a stratification on 
the moduli space of semistable G-bundles over X. 

The geometry of this stratification has been extensively studied in the case G = GL„. To 
review some of the results, first note that a topologically trivial GL„-bundle V is nothing but 
a vector bundle of rank n and degree zero, which we also denote V. For each 1 < r < n — 1, 
there is a maximal parabolic subgroup Pr of GL„ which is unique up to conjugation. A 
section a of the associated Grassmannian bundle V/Pr gives a rank r subbundle E oi V, 
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and vice versa. Since N^- = Hom(£', V/E), we get 

s{V]Pr) = m.m.{-ndegE : E CV, rk{E)=r}. 

This coincides with the well-known invariant Sr{V) of a vector bundle V, sometimes called 
the r-th Segre invariant. In the literature, a subbundle E is called a maximal subbundle of 
V if Sr{V) = -udegE. 

The first result on the invariant Sr is the upper bound 

This was obtained by Nagata [16j for n = 2 and by Mukai and Sakai |15) in general. Later, 
Hirschowitz obtained the following sharp bound [5l Theoreme 4.4] (see also [3]): 

Proposition 1.1. For a bundle V as above, we have 

Sr{V) < r{n — r){g — 1) + e, 

where < e < n and r{n — r){g — 1) + e = Q mod n. □ 

Next, let us recall the results on the stratification defined by Sr-iV). Let U{n) be the 
moduli space of semistable bundles over X of rank n and degree zero. For each integer s 
divisible by n, we define 

U{n;r,s) := {V (^U{n) : Sr{V) < s}. 
By Proposition II. 1| we have U{n;r,s) = lA{n) if s > r(n — r)[g — 1). 

Proposition 1.2. (Brambila-Paz-Lange [2], Russo-Teixidor i Bigas |19j ) For each integer 



r{n~r){g-l) 



, U{n;r,kn) is an irreducible closed subvariety oflA{n) 



k with < k < kQ : 
Also, 

codim U{n; r, kn) = r{n — r){g — 1) — kn 
for each k < k^. □ 

Moreover, each variety U (n; r, kn) can be described by using the extension spaces of fixed 
type, see [191 Theorem 0.1]. 

On the other hand, not much seems to have been studied on the properties of the invariant 
s{V; P) of a G-bundle V in general, except the following universal upper bound on s{V; P) 
obtained by Holla and Narasimhan [9l Theorem 1.1]: 

Proposition 1.3. Fix a parabolic subgroup P of G. For every G-bundle V, we have 
(1.1) s{V;P) <g-dim{G/P). □ 
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Note that when G = GL„, this coincides with the Mukai-Sakai bound discussed above, 
which is not sharp. As in the vector bundle case, the invariant siV; P) induces a stratifi- 
cation on the moduh space of semistable principal G-bundles over X. 

In this paper, we study the geometry of the stratification when G = Sp„. A vector bundle 
W is symplectic if there exists a nondegenerate bilinear alternating form w : W i^W ^ Ox ■ 
Such a bundle always has even rank 2n. It is easy to see that a vector bundle of rank 
2n is symplectic if and only if the associated principal GL2n-bundle admits a reduction 
of structure group to Sp„. A subbundle -E of Vl^ is called isotropic ii u}\e<se = 0. By 
linear algebra, the rank of an isotropic subbundle is at most n, and an isotropic subbundle 
of rank n is called a Lagrangian subbundle. A Lagrangian subbundle of W corresponds 
to a section of the associated Sp^/P-bundle W/P, where P is the maximal parabolic 
subgroup of Sp„ preserving a fixed Lagrangian subspace of C^". Let us abbreviate s{W; P) 
to SLa.g{W), where the lower case indicates "Lagrangian". Since Sp„/P is none other than 
the Lagrangian Grassmannian, the normal bundle of a section is given by Sym^^*. Recall 
that for any vector bundle V, we have 

deg(SymV) = (rk(y) + 1) deg(y), 

so 

SLsLgiW) = nim{—{n + 1) deg E : a Lagrangian subbundle of T^}. 
According to the bound (jl.ip . we get 

(1.2) s,.,iW) < 2(!^±ii,, 

In other words, every symplectic bundle W admits a Lagrangian subbundle of degree at 
least — 

One may compare this with the Hirschowitz bound on the n-th Segre invariant s„, which 
says that as a vector bundle, W admits a subbundle of half rank with degree at least 
— ["§(5 — 1)] • We prove that this slightly nicer bound is still valid for symplectic bundles. 

Theorem 1.4. For every symplectic bundle W of rank In, we have 

SLag(W/^) < \ (n(n + l)(5 - 1) + (n + \)e) , 

where < e < 1 is such that n[g — 1) + e is even. This hound is sharp in the sense that 
the equality holds for a general bundle W in the moduli space M.2n of semistable symplectic 
bundles of rank 2n over X. □ 

Next, consider the stratification on Ai2n given by the invariant SLag- For each A; > 0, let 

:= {W £ M2n : sl.s{W) < {n + l)k}. 
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By semicontinuity, A^2n ^ closed subvariety of A^2n and SLag induces a stratification 
on A42n- In particular when n = 1, since Sp^ is isomorphic to SL2, this reduces to the 
stratification already studied on the moduli space SU{2, Ox) of semistable bundles of rank 
two with trivial determinant. We prove the following result on the stratification on the 
moduli space AI4 of semistable rank four symplectic bundles over X. Note that A4^~^ is 
the whole space Ai4 by Theorem 11.41 

Theorem 1.5. For each e with I < e < g — 1, the locus Ai% is an irreducible closed 
subvariety of Ad^, of dimension 7{g — 1) + 3e. □ 

In the case of genus two, this was proven in [8]. The key ingredient of the proof there 
was a symplectic version of Lange and Narasimhan's description |14) of the Segre invariant 
using secant varieties (also see ^ for a higher rank version in the case of vector bundles). 
In this paper, we generalize the method and results of [8] to the case of arbitrary genus. We 
will see an interesting variation of Lange and Narasimhan's picture in the case of symplectic 
bundles: a relation between the invariant SLag and the higher secant varieties of certain 
fiber bundles over X. 

This paper is organized as follows. In §2, we provide the basic setup for our discussion. 
In particular, a relation between the higher secant varieties and the Segre invariants will 
be established in Theorem 12.91 In §3, we prove Theorem 11.41 using this relation combined 
with the Terracini Lemma. In §4, we study symplectic bundles of rank four in more detail. 
We will see that the relation discussed in §2 can be improved to yield a nice picture in this 
case (Theorem 14. Sp . This enables us to prove Theorem 11.51 Finally a remark will be given 
on the comparison between the two stratifications defined by S2 and SLag- 

To streamline the arguments, the proofs of two propositions will be postponed to §5; 
namely, the irreducibility of certain spaces of principal parts, and a variant of Hirschowitz' 
lemma. 

Throughout this paper, we work over the field C of complex numbers. 

2. Symplectic extensions and lifting criteria 

In this section, we establish basic results on symplectic extensions for our further dis- 
cussions. 

2.1. Symplectic extensions. Let W ^ X he a symplectic bundle and E C W a subbun- 
dle. There is a natural short exact sequence 



^ ^ W ^ E* ^ 0, 
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where E is the orthogonal complement of E with respect to the symplectic form on W. 
If is a Lagrangian subbundle of W, then E = E-^ and we get a class 

6{W) G H^{X,Rom{E*,E)) = H^{X,E®E). 

Note that we have the decomposition 

H'^{X,E0E) ^ H^{X,Sym.^E)®H\X,A'^E). 

Lemma 2.1. Assume that E is simple. An extension O^E^W^E*^0 has 
a symplectic structure with respect to which E is isotropic if and only if the class 5(W) 
belongs to H^{X,Sym^E). 

Proof. Due to S. Ramanan; see [7j §2] for a proof. □ 

2.2. Cohomological criterion for lifting. Henceforth, an extension ^ E ^ W ^ 
E* ^ will be called symplectic if W admits a symplectic structure with respect to 
which the subbundle E is Lagrangian. By Lemma l2.H this is equivalent to the condition 
6{W) G H\X,Sym.^E). 

We recall the notion of a bundle-valued principal part (see Kempf [llj for corresponding 
results on line bundles). For any bundle V over X, we have an exact sequence of Ox- 
modules 

(2.1) O^V Rat(V) Fnn(V) 

where Rat(y) is the sheaf of rational sections of V and Prin(y) the sheaf of principal 
parts with values in V. We denote their groups of global sections by Rat(y) and Prin(y) 
respectively. The sheaves Rat(y) and Prin (y) are flasque, so we have the cohomology 
sequence 

(2.2) ^ H°{X, V) Rat{V) Prin(y) ^ H^{X, V) 0. 

We denote s the principal part of s G Rat(y), and we write [p] for the class in H^{X,V) 
of p G Prin(y). Any extension class in H^(X, V) is of the form [p] for some p G Prin(y), 
which is far from unique in general. 

Now consider an extension of vector bundles 

(2.3) 0-^E-^W-^E*-^0 

and an elementary transformation F of E* defined by the sequence 

O^F-^E*^T^O 

for some torsion sheaf r. We say that F lifts to W if there is a sheaf injection F ^ W such 
that the composition F ^ W ^ E* coincides with the elementary transformation fi. We 
quote two results from [7]: 
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Lemma 2.2. [3 Corollary 3.5 and Criterion 3.6] Suppose that Hom(£;*, E)) = 

and E is simple. Let W be an extension of type h2. 3\) with class 6{W) € H^{X,E (g) E). 

(1) There is a one-to-one correspondence between principal parts p E Prin(£' (E) E) 
such that [p] = 5{W), and elementary transformations F of E* lifting to W as a 
suhhundle, given by p < — > Ker (p: E* ^ Fnn (E)) . 

(2) The suhhundle corresponding to Ker(p) is isotropic in W if and only if^p = p. □ 

2.3. Subvarieties of the extension spaces. Let V hea. vector bundle with V) ^ 0. 

We describe a rational map of the scroll ¥V into the projective space ¥H'^{X,V). Let 
vr: W —7- X be the projection. We have the following sequence of identifications: 

H^{X,V) ^ H^{X,Kx®V*Y 

^ H\X,Kx®^.O^v{l)Y 

^ H\X,'n,{7i*Kx®0^v{lW 

Hence via the linear system \Tr*Kx C'pv(l)|, we have a map tp: FV --^ FH^{X, V). For 
V = Rom{E*,E) = E (g) E, we get a map 

if : F{E ® E) FH^{X, E (g) E). 

Also for V = Syvc?E, we get a map 

PSym^^ --^ FH^{X, Sym^E), 

which is nothing but the restriction of to PSym^i?. If w E V is nonzero, we will abuse 
notation and let v also denote the point of FE corresponding to the vector v. By restricting 
9? further to the subvariety FE C PSym^i? embedded by the map v i— ?> u (8) f , we get 

il^-.FE --^ FH^{X, Syrn^E). 

Now we give another way to define ip and ip, which will be convenient in what follows. 
This idea was used by Kempf and Schreyer [12l §1] to define the canonical map X \Kx\'^ , 
and generalized in [U §3.3]. 

For each x E X, we have an exact sequence of sheaves 

V(x) 

O^V ^ V{x) ^ 0, 

which is a subsequence of (|2.ip . Taking global sections, we obtain 

(2.4) ^ H^{X, V) H^iX, V{x)) V{x)\^ ^ H^{X,V) ^ ■■■ 

The restriction of ip to the fiber FV\x is identified with the projectivized coboundary map d 
in (|2.4|) . In view of (|2.2|) , it can be identified with the map taking v E FV to the cohomology 
class of a F-valued principal part supported at x with a simple pole in the direction of v. 
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Similarly, tp can be identified with the map taking v G E\x to the cohomology class of a 
Sym^E'- valued principal part supported at x with a simple pole along u (g) v. 

Lemma 2.3. Consider the above rational maps (p and tp for a vector bundle E. 

(1) The map (p is an embedding if E is stable and ^{E) < —1. 

(2) If E is stable with ^{E) < —1/2 then if is base-point free. 

(3) If g > 3 and E is general in h({2, d), d < 0, then ip is an embedding. 

Proof. (1) For any divisor D on X, write V{D) for the bundle V (8> Ox{E>)- One can 
easily see that ip is base point free if H'^{X, V{x)) = for all x e X, and an embedding if 
H^{X,V{D)) = for all effective divisors D of degree 2. Since E is stable, so is E*, and 
the condition fi{E) < — 1 implies that 

li{E*) > n{E) + 2. 

If there were a nonzero map a: E* ^ E{D) for some D G Syra^X, then we would have 
n{E) + 2 < fi{E*) < /x(Ima) < ix{E{D)) = fi{E) + 2, 

which is a contradiction. Thus H^{X,Rom{E* , E{D))) = 0. 

(2) The same argument as above shows that if E is stable and fJ-{E) < —1/2, then 
H^{X, V{x)) = for all x G X. Therefore, in this case if is base-point free. 

(3) By (1), ip is an embedding for d < —1. Now suppose d = —1. By (2), we know that 
ip is base-point free, and hence so is 

To show that ip is an embedding, it suffices to show that for any x,y & X, 

• No two principal parts of the form 

z w 

arc identified in H^{X, Sym^E) for any nonzero e G E\x and / G E\y, where z and 
w arc local coordinates at x and y respectively, and 

• the principal part 

e (g) e 

is not cohomologically zero for any nonzero e G E\x. 
The first statement is equivalent to saying that there does not exist a symmetric map 

a: E* E{x + y) 



with principal part equal to 



e (g) e / (g) / 
z w 
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First assume that a is generically injective. Note that any such a has rank one at x and 
y. Hence the non-zero section det(a) of the hue bundle 

Hom(det E*,det E{2x + 2y)) ^ (det Ef{2x + 2y) 

vanishes at x and y. Thus {det E)'^ (x + y) is trivial, so (detS)~^ is effective. But if 5 > 3, 
a general E gU{2, —1) does not have this property. 

Next, assume that a is of generic rank 1. Then a factorizes as 

E* ^ M ^ E{x + y) 

for some rank 1 sheaf M of degree m. The surjection E* ^ M and the injection M — >■ 
E{x + y) imply that E admits a line subbundle of degree at least 

max{deg M*, deg -/Vf(— X — y)} = max{— m,m — 2} > — 1. 

This shows that E is not general in U{2, —1) for g > 3 by a known property of the Segre 
invariant of E, which is analogous to Proposition ll.il For a rank 2 bundle E of degree d, 
define 

si{E) := mm{d — 2degL} 

where L runs through all the line subbundles of E. Then by Lange-Narasimhan |14^ 
Proposition 3.1], we have 

(2.5) g-l<siiE)<g 

for a general E € U{2, d). This in particular shows that for g > 3, a general E G U{2, —1) 
does not admit a line subbundle of degree > — 1. 

Hence we conclude that there is no such symmetric map a. This proves the first state- 
ment. The second statement can be proven by a similar argument with x = y. □ 

2.4. Geometric criterion for lifting. Firstly we adapt some ideas from [F, §3] to the 
present situation. 

Definition 2.4. Let V ^ X he a vector bundle. Then an elementary transformation 

is called general if r = 0(L]^ C^;. for distinct points xi, . . . ,Xk of X. 

The word "general" is justified as follows. As in [3, §3], one can consider a parameter 
space Qk{V) of elementary transformations of V of degree deg(y) — k and observe that the 
locus of general elementary transformations is open and dense in Qk{V). 

Definition 2.5. Let y be a closed subvariety of a projective space F^. The k-th secant 
variety of Y, denoted by Sec'^y, is the Zariski closure of the union of all the /c-secants to 
Y, that is, all the projective subspaces of spanned by k distinct points of Y. 
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Definition 2.6. For each x G X, we denote A|^ the projectivization of the set of all rank 
one linear maps E*\x E\x- The union of all the A\x is a fiber bundle A inside the scroll 
¥{E (8> E), which we call the decomposable locus. We will also use the fiber subbundle of A 
given as the image of the Veronese embedding ¥E ^ ¥{E E). 

Now we will consider a bundle E ^ X of rank n which satisfies either condition (1) or 
the condition (3) in Lemma 12.31 so that either 

^p■.F{E^E)^FH^{X,E0E) or : FE ^ FH^iX^Sym^E) 

is an embedding. Consider the following diagram: 

A > F{E(E)E) > FH^[X,E ®E) 



FE y P(Sym2^) y FH\X,Sym^E). 

If ^ is an embedding, the composition of the two maps in the bottom row is an inclusion, 
whereas if is an embedding, all the arrows are inclusions. 

Lemma 2.7. Consider a bundle W fitting into a nontrivial symplectic extension of E* by 
E with class 5{W) £ FH\X,Sym'^E). 

(1) // W admits an isotropic lifting of an elementary transformation F of E* with 
deg{E*/F) < k, then 5{W) belongs to Sec'^FE. 

(2) // the class 6{W) corresponds to a general point of Sec^FE, then W admits an 
isotropic lifting of a general elementary transformation F of E* with deg{E*/F) < 
k. 

Remark 2.8. An analogous lifting condition for the map (p was considered in [31 The- 
orem 4.4], where a criterion on the lifting of an elementary transformation of E* with- 
out the isotropy condition, was given in terms of the higher secant variety of the scroll 
A C FH^iX, E(^E) instead of FE C FH^{X, Syrn^E). 

Proof. Basically we follow the same idea underlying the proof of [31 Theorem 4.4], except 
that we use the language of principal parts here. 

By Lemma 12.21 (2), an elementary transformation F of E* lifts to W isotropically if and 
only if the class 5{W) = [p] for some symmetric principal part p such that 

F C Ker {p: E* ^ Prin(S)) . 

First assume that F is a general elementary transformation so that p is a linear combination 
of k symmetric principal parts pi, . . . ,pk supported at distinct points xi, . . . , x^, with 
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for each i. By symmetry, each pi belongs to A|j.. nPSym^E'l^^j = IP-E'|x'i- By our alternative 
definition of ip: F{E (gi E) ^ FH^{X,E(g) E) at the end of g231 the point 5{W) = \p] lies 
on the secant plane spanned by k distinct points of Pi?, in other words, (5(VF) G Sec'^(P£'). 

The proof of (1) will be completed by taking limit, once we know the irreducibility of the 
space of symmetric principal parts defining elementary transformations of degree deg E* — k. 
The detailed proof of this rather obvious-looking fact will be given in §5.1. 

To show (2), suppose 5{W) lies on a secant plane spanned by k general points of ¥E. 
This means that 6{W) can be defined by a linear combination p = ^ \iPi of at most k 
symmetric principal parts pi supported at distinct points, where pi G PE^I^;.. By Lemma 
12.21 (2). the kernel oi p: E* ^ Prin(£') lifts to W isotropically. Also, Ker(p) is a general 
elementary transformation of E* whose degree is at least deg-E* — k. □ 

From the above discussion, we obtain: 

Theorem 2.9. Let E and W he as above. If S{W) G Sec'^FE, then we have SLag(W) < 
(n + l)(A: + deg£;). 

Proof. By Lemma 12.71 (2), if 6{W) is a general point of Sec'^PE', then there is some ele- 
mentary transformation F <Z E* lifting to W isotropically such that deg(F) > degE* — k. 
Hence by definition, SLag(l^) ^ (n + l){k -\- degE). By the semicontinuity of the invariant 
SLag, this inequality stih holds for any W with 5{W) € Sec'^PE'. □ 

Remark 2.10. (1) In [3l Theorem 4.4], it was proven that if 6{W) e Sec'^A then we have 
SniW) < 2n(/c + deg E), that is, W contains a rank n subsheaf of degree deg(£'*) — k, which 
is not necessarily isotropic. We will return to this phenomenon in ^4.31 

(2) One can ask if the converse holds in Theorem 12. 91 This is a subtle question. Certainly 
the condition SLag(^) < (^^ + l)(A; + deg£') implies that W admits an isotropic subbundle 
F of rank n and degree > degE* — k. But in general F need not come from an elementary 
transformation of E* , due to the possible existence of the diagram of the form 

> E > W y E* y 

(2.7) j j 

> G > F y H y 

where rk(G) > 1. In other words, for n > 1, it can happen that a maximal Lagrangian 
subbundle of W may intersect S in a nonzero subsheaf. But in §4.1, we will show that this 
kind of diagram appears only in a restricted way for n = 2. 

3. Upper bound on SLag 

One can easily see that the invariant SLag has no lower bound by considering, for instance, 
a decomposable bundle E (B E* where E is a bundle of arbitrarily large degree. In this 
section, we prove Theorem 1 1 . 4 1 which gives us the sharp upper bound on SLag- 



LAGRANGIAN SUBBUNDLES OF SYMPLECTIC BUNDLES 



11 



In [3], the Hirschowitz bound in Proposition [TTT] was reproved using the relation between 
the invariant Sr and the geometry of the higher secant varieties of the ruled varieties in the 
extension spaces. We would like to adapt the same idea to the case of symplectic bundles 
by applying the results in the previous section. We begin with the following observation. 

Lemma 3.1. A general symplectic bundle W € A^2n has a Lagrangian subbundle E of 
degree < —^n{g — 1) which is general as a vector bundle. 

Proof. First we show that every symplectic bundle W has a Lagrangian subbundle. Let 
U C X be an open set over which W is trivial. Any Lagrangian subspace of C^" yields a 
Lagrangian subbundle of This corresponds to a rational reduction of structure group 

of the associated Sp„-bundle of W to the maximal parabolic subgroup preserving a fixed 
Lagrangian subspace. Since X has dimension one, this can be extended to a reduction of 
structure group over the whole of X, giving a Lagrangian subbundle of W (cf. Hartshorne 
[H Proposition L6.8]). 

Now let Wq be a general symplectic bundle. Since Wq admits a Lagrangian subbundle, 
say Eq, we have an extension 

0^ Eo^Wq^ E^^O 

with class 5{Wo) G H^{X, Sym^ii^o)- Now it is well known that there is a versal deformation 
£ S X X oi Eq parameterized by an irreducible variety S, whose general member is a 
general stable bundle. Consider the direct image -R^g*(Sym^£^), where q: S x X ^ S is the 
projection. This sheaf is locally free over the open subset 5*° of S consisting of the bundles E 
with h^{X, Sym^^;) = 0. We claim that Eq belongs to S°: Indeed, a nonzero map Eq — >■ Eq 
would induce an endomorphism of W given by composition Wq Eq ^ Eq ^ Wq. This 
is clearly nonzero and nilpotent. But this contradicts the stability of W. 

The associated projective bundle ¥{R^q^:{Sym'^£)\s°) over 5*° will be denoted vr : P ^ S°. 
By Lange [1'6\ Corollary 4.5], there is an exact sequence of bundles over P x X: 

^ (vr X ldx)*£ P*Op{l) ^ W ^ (vr X IdxTS* ^ 0, 

where p: P x X — )■ P is the projection, with the property that for each 6 in the fiber 
-K^^([E]) in P, the restriction W|{5}xx is isomorphic to the symplectic extension of E* 
by E defined by 6. From the construction of 5°, a general point of P corresponds to a 
symplectic bundle admitting a Lagrangian subbundle which is general as a vector bundle. 

Finally we check that deg{E) =: —e < —^n{g — 1). From the above argument, the 
dimension of the space of symplectic bundles admitting a Lagrangian subbundle of degree 
— e is bounded above by 

dimU{n,-e) + h\X,Sym^E) -1 = n'^{g - 1) + 1 + {n + l)e + ^n{n + l){g - 1) - 1. 
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Since this should be at least dim A^2n = n{2n + l){g — 1), we obtain e > ""^^^ . □ 

To prove Theorem ll.4l it suffices to show that a general W € A42n has a Lagrangian sub- 
bundle of degree at least — — 1)] • By Lemma [3.11 we know that W has a Lagrangian 
subbundle E which is general in U{n, — e). 

By Lemma 12.31 (1), we may assume the map ip: F{E ^ E) ^ FH'^{X, E (g> E) is an 
embedding for sufficiently large e. Consider its restriction ip: ¥E — )■ P := ¥H^{X, Sym^E). 
By Theorem 12.91 if the class 5{W) E P belongs to the subvariety Sec^¥E, then W has a 
Lagrangian subbundle of degree at least e — k. For the moment, suppose that ¥E has no 
secant defect in ¥ so that for each > 1, we have 

dimSec'^PS = min{/c dimP£; + A: - 1, dimP} 

= min{/c(n + 1), (n + l)e + — n(n + — 1)} —1. 

By straightforward computation, Sec'^P-E = F ii k > ko := e + I'^n^g — 1)] . This implies 
that W has a Lagrangian subbundle of degree at least — [^n(5( — 1)] , as was desired. 
Hence the proof will be completed once we show the following. 

Proposition 3.2. For a general bundle E G l{{n, —e), e 3> 0, the subvariety FE o/P := 
¥H^{X,S'ym'^E) has no secant defect. □ 

The rest of this section is devoted to the proof of Proposition 13.21 First we recall the 
Terracini Lemma |21| : 

Lemma 3.3. Let Z C P^ be a projective variety and let zi, . . . , Zk be general points of Z. 
Then dimSec'^Z = dim{Tzj^Z, . . . ,Tzf.Z) , where Tz.Z denotes the embedded tangent space 
to Z in P^ at zi. □ 

To apply the Terracini Lemma, let us find a description of the embedded tangent spaces 
of FE C P. Let be a nonzero vector of E\x^ and consider the elementary transformation 
0— T'ii^^E'^Ci^^O such that the kernel of E\x E\x is spanned by v. Consider the 
induced elementary transformation 

^ Sym^E; ^ Sym^E ^ r ^ 

where r is a torsion sheaf of degree (n + 1). 

Lemma 3.4. For a general bundle E S h{{n, —e), with e ^ 0, the embedded tangent space 
to FE at V inF is given by 

T^{FE) = PKer Sym^EJ) ^ H^{X,Sym^E)^ . 
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Proof. Recall that A is the subvariety of F{E E) C FH^{X, E E) consisting of decom- 
posable vectors. We have the commutative diagram 

H'^{X,E®E) H^{X,E^E) y 

(3.1) I 

H^{X,Sym'^E) H^{X,Sym'^E) > 0, 

where the vertical arrows are inclusions. By [3l Lemma 5.3], the embedded tangent space 
of A at f (gjf in FH^{X,E®E) is given by T^^^A = P(Ker/). From the inclusion P£' C A 
and the above diagram ()3.ip . we get 

T^FE C T^^^A n FH^{X,Sym^E) = PKer(/) n FH^{X,Sym'^E) = FKer{g). 

Thus it suffices to show that FKer{g) has dimension n = dimPii^. Since E is general, we 
may assume E is stable. Then Sym^E' and Sym^E are semistable of negative degree, so 
they have no nonzero sections. Thus Ker (51) has dimension 

h^{X,Sym.^E)-h^{X,Sym^E) = deg{Sym.^ E) - deg{Sym.^ E) = n + 1, 

as desired. □ 

Proof of Proposition \3.2[ Let F be the general elementary transformation of E associated 
to the points vi, . . . ,Vk € FE. In other words, F fits into an exact sequence 

k 

(3.2) O^F^F^0C^^ ^0, 

for k distinct points xi, . . . ,Xk such that Vi lies over Xj. For each i, let Ei be the intermediate 
sheaf {E C Ei C F) defined by the elementary transformation associated to the Vi: 

^ E ^ Ei ^ C:,^ ^ 0. 

By the Terracini Lemma and Lemma 13. 4^ we see that the dimension of the embedded 
tangent space of FE CF at Vi is given by the dimension of the linear span of the union of 

(3.3) PKer {H^ (X, Sym^S) ^ {X, Sym^ Ei)) 

for 1 < i < k. Since F is precisely the intersection of all the Ei inside E, the linear span of 
the spaces (|3.3p coincides with 

PKer {H^iX, Sym^E) H^{X, Sym^F)) . 

Thus now it remains to show 

dim Ker {H^ {X, Sym^F) ^ {X, Sym^F)) = min{A;(n + 1) , (n + l)e + ^n(n + 1) (5 - 1) } . 

From the vanishing of H^{X, Sym^F), one checks that the left hand side equals k{n + 1) — 
h^{X, Sym^F). Since F is obtained from a general elementary transformation of a general 
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bundle E, it is a general element of U{n, —e + k). Hence the required equality follows from 
the result on dim //'^(X, Sym^F) which will be discussed in Lemma 15.31 and Corollary 15.51 
in §5.2. □ 

Remark 3.5. (Hirschowitz bound for Gp„-bundles) 

The statement of Theorem 1 1.41 is easily generalized to principal Gp„-bundles. Recall that 
the group Gp„ of conformally symplectic transformations is the image of the multiplication 
map Sp„ X C* — 7> GL2n- A principal Gp„-bundle corresponds to a vector bundle of rank 
2n carrying a symplectic form with values in a line bundle L which may be different from 
Ox] equivalently, admitting an antisymmetric isomorphism W — > W* (8i L. Such W has 
determinant L" and hence degVl^ = ndeg(-L). (See Biswas-Gomez [IJ for more details on 
Gp„-bundles.) 

If C is a Lagrangian subbundle then, by [3 Criterion 2.1], we get an extension 

E -^W Hom(^, L) ^ 

with class 6{W) G H^{X, L^^ ^ Sym^E), and conversely. Arguing as above, one can show 
that W admits a Lagrangian subbundle of degree at least — {g — 1 — deg(L))] . 



4. Geometry of the strata in rank four 

In this section, we prove Theorem 11.51 on the geometry of the strata on A^4. For the 
case of genus 2, it has already been proven in [8]. Throughout this section, we assume 
g>3. 

4.1. Symplectic extensions of rank four. First we study more details on symplectic 
extensions of rank four. The goal is to prove the converse of the statement in Theorem 12.91 
in the case of rank four. As was mentioned in Remark 12. lOf 2). we need to know how often 
diagrams of the form (|2.7p appear. 

For e G {1,2, . . . , g — 1}, let —t- X be a vector bundle of rank two and degree — e. 
Assume that E is general, so — 1 < si{E) < g as was remarked in (|2.5|) . Equivalently, 
assume that any line subbundle of E has degree at most ^{—e — g + 1). 

Lemma 4.1. Let E be as above and consider an extension O^E^W^E*—^0 (not 
necessarily symplectic). Let F be a rank two subbundle of W such that the intersection of 
E and F is generically of rank one. Then 

(1) degF< -(5-1). 

(2) If degE = degF = —{g — 1), then the intersection of E and F is a line subbundle 
of W of degree —{g — 1). 
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Proof. (1) Let L be the line subbundle of W associated to the intersection of E and F. By 
the assumptions, we have a diagram 

> E > W > E* > 

> L > F > M > 

where M is a subsheaf of E* . Since E is general, 

deg(L) < ^(-e-5 + 1) and deg(M) < ^(e - 5 + 1). 
Hence deg(F) = deg(L) + deg(M) <-{g-l). 

(2) By the above inequalities, the condition deg-E = degF = —{g — 1) implies that 
deg-L = —{g — 1) and degM = 0. If the quotient sheaf E* /M had nonzero torsion, then 
E* would admit a quotient line bundle of degree < g — 1, contradicting the generality of 
E. Hence M must be a line subbundle of E* , which shows that the intersection of E and 
F coincides with L. □ 

The following is an immediate consequence of Lemma l4.ll (1). 

Corollary 4.2. Let E he a rank two bundle of degree —e with 5 — 1 < si{E) < g. Consider 
a symplectic extension ^^E^W^E*^{) corresponding to a general point 5{W) of 
¥H^{X, Sym^E). 

(1) Ifl<e<g — 2, then any Lagrangian subbundle ofW other than E itself, of degree 
> —e, comes from an elementary transformation of E* . 

(2) If e = g — 1, then any Lagrangian subbundle ofW of degree > —e comes from an 
elementary transformation of E* . □ 

Prom this, we get a nice geometric criterion on lifting which improves Theorem 12 .91 in the 
case of rank 4. Recall the embedding criteria on of Lemma 12.31 (3), confirming that for 
g >3 and for a general bundle E e W(2, -e) with e > 0, the map ip-.FE-^ FH'^{X, Sym^E) 
is an embedding. Hence in this case, we get a surface Tp{FE) = FE inside FH'^iX, Sym^E). 

Theorem 4.3. Assume g > 3 and 1 < e < 5— 1. Let E be a general bundle inU{2, —e), and 
consider a nontrivial symplectic extension W of E* by E. For each k with 1 < k < 2e — 1, 
the following conditions are equivalent: 

(1) W admits an isotropic lifting of an elementary transformation F of E* with degF > 
e — k. 

(2) 5{W) e Sec'^FE in FH^{X,Sym^E). 

(3) SLagW < (n + l)(fc-e). 
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Proof. The implications (1) (2) and (2) =^ (3) were already shown in Lemma 12.71 (1) 
and Theorem 12.91 respectively. The implication (3) =^ (1) can be readily seen as follows. 

The condition SLag(W^) < {n + l){k — e) implies that W admits a Lagrangian subbundle 
F of degree at least e — k. By Corollary 14.21 if degE = —e < e — k < deg-F, then F comes 
from an elementary transformation of £^*. □ 

4.2. Stratification on M4. For any positive integer e, let U{2, — e)* be the moduli space 
of stable bundles over X of rank 2 and degree — e. According to Narasimhan and Ramanan 
[TTI Proposition 2.4], there exist a finite etale cover t:^'- Ue ^(2,— e)* and a bundle 

— > f/e X X with the property that £e\{E}xX — '^eiE) for all E ^ Ue (for odd e, we can 
take TTg to be the identity map since U{2, — e) is a fine moduli space). 

Now by Riemann-Roch and semistability, for each E G U{2, —e)^, we have 

dimH^{X,Sym'^E) = 3e + 3{g-l). 

Therefore, the sheaf i?^p*Sym^(£'e) is locally free of rank 3(e + g — I) on Ue- Consider its 
projectivization /i: Pg — )■ C/g- 
Now we have a diagram 

X 




We write r 



X X 



U, 



for the projection. By Lange [13, Corollary 4.5], there is an 



exact sequence of vector bundles 

^ (/i X IdxYSe O r*Op^(l) ^We^ifiX IdxTS: ^ 

over PgXX, with the property that for (5 G Pg with fi{5) = E, the restriction of We to {6}xX 
is isomorphic to the symplectic extension of E* by E defined by 5 G FH^{X, Syni^E). 

There arises a basic question regarding these extension spaces: Is a bundle W correspond- 
ing to a general point 5 G FH^{X, Sym^E) stable, if E is taken to be general? (The same 
question for the vector bundles was answered affirmatively by Brambila-Paz and Lange [2] , 
and Russo and Teixidor i Bigas [IHj.) The machinery obtained in the previous subsection 
enables us to answer this question. 

Lemma 4.4. Let E G h{{2, — e), 1 < e < g — 1, be a general bundle such that g — 1 < 
si{E) < g. Then a general point of FH^{X,Sym.'^E) corresponds to a stable symplectic 
bundle. 
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Proof. By Theorem ll.4l a general symplectic bundle W in Ai^ satisfies SLag(^^) = •iig — !)• 
This shows the above statement for e = g — 1. 

Now assume 1 < e < g — 1. Let W he a symplectic bundle corresponding to a general 
point of ¥H^(X,Sym^E). Consider any line subbundle L of W. If L is contained in the 
subbundle E, then deg(L) < by the stability of E. Otherwise L would yield an invertible 
subsheaf of E* via the composition L W —j' E* . From the condition on si{E) = si{E*), 
we have 

degL<^^<0 

by generality of E. Therefore, any line subbundle of W has negative degree. Since sym- 
plectic bundles are self-dual, the same holds for subbundles of rank three. 

Finally we consider the Lagrangian subbundles of W. By Theorem 14.31 SLag(W^) < if 
and only if 6{W) G Sec'^P^. But 

dimSec'^PS < 2e (e - 1) = 3e - 1, 

while dim.¥H^{X, Sym^E) = 3e -|- 817 — 4. Therefore, the bundles with SLag(W^) ^ form 
a proper subset mFH^{X, Sym'^E) . □ 

Now we go back to the diagram (j4.2p . There is a classifying map 7e : Pe A^4 induced 
by the bundle We ^ Pe x X. This map je is defined over a nonempty dense subset of Pg. 

Recall the definition of the stratification on given by the invariant SLag^ for each 
e > 0, consider the subvarieties of A^4 defined by 

Ml := {W eMi : SLag(VF) < 3e}. 

Now we can show the following result which implies Theorem 11.51 



Theorem 4.5. (1) For each e with 1 < e < g — 1, the map je is generically finite and 
its image is dense in A^|. 

(2) For each e with 1 < e < 5 — 1, the locus M% is irreducible of dimension 7[g—l) + 2>e. 

(3) For e < g — 2, a general point of M% corresponds to a symplectic bundle which has 
a unique maximal Lagrangian subbundle. In particular for odd e < g — 2, the locus 
Ml is birationally equivalent to the fibration Pg overlA{2, —eY. 

Proof. First consider the case e = g — 1. From the bound on SLag given in Theorem [T31 we 
have A^4 = ^. In particular, M\ ^ is irreducible of dimension 10(3 — 1). Moreover, 
the fact that a general symplectic bundle W satisfies SLag {W) = 3{g - 1) implies that 
7g_i : Pg-i — M4 is dominant. But dimPg_i = 10{g — 1), so 7^-1 must be generically 
finite. 

In general, it is clear that the image of 7e lands on the stratum M^. Since the source Pe 
is irreducible, so is the image of 7e. Now we show that it is dense in M^. Any W in M% 
is fitted into a symplectic extension of E* by E for some E of degree — e which might be 
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unstable. But every such E is contained in an irreducible family of bundles whose general 
member is a stable bundle in U{2, — e). This shows that is the closure of the image of 
7e. In particular, is irreducible for each e. 

Now assume 1 < e < g — 2 and consider a point of Fg, that is, a symplectic extension 

where E G U{2, — e)* is general, so si{E) > g — 1. Suppose that W admits a Lagrangian 
subbundle F of degree > — e other than E. Then by Corollary 14.21 (1), F is an elementary 
transformation of E* which lifts to W. By Theorem 6{W) £ Sec^^'iFE). But 

dimSec2^(P£;) < 6e - 1 < 3e + 3g - A = dim¥H^{X,Sym^E). 

Thus an extension represented by a general point of Pel^; contains no Lagrangian subbundle 
of degree > — e apart from E itself. This implies that a general symplectic bundle in 7V4| 
has a unique maximal Lagrangian subbundle. In other words, W is represented only in the 
fibers of Pg Ue over the finite subset 7r~^{E) for the etale cover vrg : f/g ^ ^(2, — e)*. 
Moreover, there is only one extension class in P//^(X, Sym^i?) whose associated bundle is 
isomorphic to W. 

This shows that je is generically finite onto its image, of degree equal to that of vTe- In 
particular, for odd e < 5 — 2, 7e is generically injective since vrg is the identity map, and so 
is birationally equivalent to the fibration Pg over h{{2, — e)*. Also, for all e we obtain 
dim7W| = dimPe = 7{g-l) + 3e. □ 

4.3. Nonisotropic maximal subbundles. By Serman [20], for n > 1 the forgetful map 
taking a symplectic bundle to the equivalence class of the underlying vector bundle gives 
an embedding of A42n in the moduli space Sl/l{2n,Ox) of semistable bundles of rank 2n 
with trivial determinant. Thus it is interesting to compare the two stratifications on A^2ra 
given by s„ and SLag- We will focus on the rank four case. 

Suppose X has genus 5 > 4. Let Fi and F2 be a pair of mutually nonisomorphic stable 
bundles of rank two and trivial determinant. The direct sum W := Fi(B F2 is a symplectic 
bundle with S2{W) = and SLag(l^) > 0. We will now use Lemma [2771 and Remark 12.81 to 
illuminate this phenomenon. 

Let E he a general bundle of rank two and determinant Oxi—x) for some point x G X. 
The constant function 1 defines a global rational section a of det{E) which has a simple 
pole at X and is elsewhere regular. Since E has rank two, for any linearly independent 
v,w £ E\x we have, up to a constant, 

V f\w V ®w w (^v 
(4.1) a = = . 

z z z 
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Thus the cohomology class 



V <Si w 




W ^ V 


z 




z 



in H^(X, E E) defines a symplectic extension of E* by E. 

Write p = By Lemma O (2), the map A ^ '^H'^{X, E (g) E) is base point free. 
Since [p] = ip{v (X" w), it is a nontrivial cohomology class. 

By Lemma [2.21 the kernel F of p: E* — )• Prin(£') lifts to a nonisotropic subbundle of W. 
This subbundle is an elementary transformation — )• -F — )• -E* — )• C^^ — > 0, so has trivial 
determinant (as it must, since h^{X, A^F*) > 0). 

This behaviour can be explained geometrically as follows. Since 



(4.2) 







W ^ V 


z 




z 



for any linearly independent v and w, the map ip: A\x ¥H^{X,E ® E) is not an embed- 
ding. In fact it is a double cover 

A|a: ^ X ^ PSym^E;]^ ^ P^ 

ramified over the plane conic P-E|a;. (In particular, it factorizes via Pi7"'^(X, Sym^i?).) By 
(j4.2p . any extension class lying in this P^ lies on (a 1-secant to) the quadric bundle A, but 
a general such class does not lie on (a 1-secant to) the conic bundle Pi?. Therefore, by 
Lemma |2.7| an elementary transformation of degree deg(£'*) — 1 = lifts to a nonisotropic 
subbundle of W , but W has no Lagrangian subbundle of degree zero. 

Remark 4.6. In fact W admits a pair of nonisotropic subbundles of trivial determinant 
given by Ker(p) and Ker(*p). It is not hard to see that these are of the form F and F-^. 
Furthermore, the bundle W splits as the direct sum F © F^, by [6, Theorem 2.3]. 

Building upon this idea, one can show the following: 

Suppose 5 > 4. For any e and f with < f < e and / + 2e — 1 < ^^-^^"^^ , there ex- 
ists a stable rank four symplectic bundle W with S2{W) = 4/ and SLag(l^) = 3e. In other 
words, W has a maximal subbundle of degree —f which is nonisotropic, and a maximal 
Lagrangian subbundle of degree —e < —f. 

The principle is as above: for certain E of degree — e and special determinant, one can 
construct a class 6 in H^{X,Sym^E) lying on Sec''+^ A \ Sec^^"^P£;. By Lemma [121 
the extension O^E^W^E*^0 defined by 5 will have a nonisotropic subbun- 
dle of rank two and degree — /, but no Lagrangian subbundle of degree greater than — e. 
The details are tedious, so we omit the calculation. We mention only that the condition 
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/ + 2e — 1 < ^^^„ comes from Hwang-Ramanan jlOI Proposition 3.2], which guarantees 



3 

the vanishing of H^{X, adE{D)) for certain effective divisors D. 

5. Proofs of two lemmas 

In this miscehaneous section, we provide proofs of two lemmas which were used in the 
previous sections. Both statements may look rather obvious, but we could not find shorter 
proofs than those given here. 

5.1. Irreducibility of spaces of symmetric principal parts. Let be a vector bundle 
over X of rank n. For a principal part p £ Fnn{E), we define the degree of p as the 
degree of the torsion sheaf Im (p : E* — ?> Pnn (E)). A principal part of degree d is called 
general if it is supported at d distinct points of X. Clearly, the elementary transformation 
Ker (p : E* ^ Fnn (E) ) of E is general in the sense of Definition 12.41 if and only if p is 
general. 

In this subsection, we prove the following. 

Proposition 5.1. Every symmetric principal part p € Pnn{E (S) E) is a limit (in the 
analytic topology) of a sequence of general symmetric principal parts of the same degree. 

Proof. It suffices to consider the case where p is supported at a single point x € X. In this 
case, we define the order of p to be the smallest integer k such that Im(p) is contained in 
E{kx)/E. Let z be a local coordinate at x and let ei,e2, . . . ,e„ be a frame for E near x. 
Then the symmetric principal part p of order k is locally expressed as 

where ipi is a symmetric tensor expressed by 

11^1= ^ a\j{ei (g) ej + ej (g) Cj). 

l<i<j <n 

Now we claim that after a suitable change of the local frame, every symmetric principal 
part p of degree d and order k can be expressed by 

r 

p = Pi where pi 



e,: (» e,; 



i=l 



and k = ki > k2 > ■ ■ ■ > kr, and Yll=i ^« ~ ^- easy to see from this expression that p 
is a limit of a sequence of general symmetric principal parts of degree d: For each i, choose 
ki distinct complex numbers ci , C2 , . . . , Cfc . and consider the 1-parameter family of principal 
parts 

p.^f"^ (g) ej 

{Z - Clt){z - C2t) ■ ■ ■ {Z - Ck^t)' 
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For t 7^ 0, each pi{t) is a general principal part of degree ki, and J2i=i Pi{^) = P- Thus p is 
the limit of a sequence of general symmetric principal parts of degree Yll=i = ^• 

Now we prove the claim. Since V'fc is symmetric, we may assume after a change of frame: 

V'fc = ei ® ei + 62 (8) 62 H \res®es 

for some s < n. Now we make a change of local frame given by 

n 

e[ = ei + z'^a'ljej. 

Then it is straightforward to check that p is expressed as 

e'l e[ ^ _^ 4>k-i _^ ^01 

where (f)k,(l>k-i e Sym^ < 62, 63, • • • , > and (j)k-2, 0ik-3, ■■■ e Sym^ < e^, 62, • • • , e„ >. 
For 

n 

<t>k-2 = '^bijie'i® Cj + Cj e'l) + ^ 6ij(ei (g) ej + (g) e^), 

j=l 2<i<j<n 

we make a change of local frame given by 

n 

i=2 



Then we can check that p is expressed as 



e'l ^ e'l I (t>'k I '^fc-i I I ^1 



where ^'j., (\>'^_x, G Sym < 62, 63, ■ ■ ■ , e„ > and (;li'^_3, ■■■ ^^'^e Sym < e'/, 62, ■ ■ ■ , e„ >. 
Continuing this process, we finally get an expression for p of the form 

ei ® ei 
fc '"5' 

where q' is a symmetric principal part expressed in 62, . . . , e„. Next we perform the same 
operation on q and 62- Continuing this process, finally we get an expression for p of the 
form 

_ 61 (8) 61 ^ 62 (8> 62 ^ 6r (g 6r 

as required. □ 



Corollary 5.2. For a fixed vector bundle E over X and for each d > 0, the space of 
symmetric principal parts in Prin(£^ ® E) of degree d is irreducible. 
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5.2. A variant of Hirschowitz' Lemma. Hirschowitz' lemma [S| 4.6] ensures us that 
the tensor product of two general bundles is nonspecial (see also Russo-Teixidor i Bigas 
[19\ Theorem 1.2]). This is useful in many situations; for instance the proof [3], p. 12] of 
Hirschowitz' bound on the Segre invariants. Here we will prove a variant of Hirschowitz' 
lemma. 

Lemma 5.3. Let F be a general stable bundle of rank n and degree d. If d < ^n{g — 1), 
then h^{X,F(^F) = 0. 

Proof. The proof will be completed in three steps. 

Step 1. Let F be a deformation of F given by a nonzero class 5 G End(-F)). For a 

given nonzero symmetric map a : F* —^F,we want to know when there exists an extension 
d : F* F inducing the following commutative diagram: 

> F > ¥ > F > 

(5.1) 

> F* > ¥* y F* > 0. 

We have induced maps 

a^: H^{X,End{F*)) ^ H^{X,}lom{F* , F)) 

and J: i?^(X, End(F)) ^ H^{X,Rom{F\ F)). 

Note that the class of the dual deformation F* is given by —^6 € H^{X,Find{F*)). By 
straightforward computation, we can check that the maps a on the outer terms in the 
diagram (|5.ip extend to a map d: F* ^ F if and only if —a\, *5 = in H^{X, Hom(F*, i^)). 
Now since a is symmetric, we have 

-ab*5 = -(*a)/(5 = -*(a«<5). 

Thus we obtain: 

Lemma 5.4. A nonzero symmetric map a : F* ^ F extends to d if and only ifa^5 belongs 
to the subspace H^{X,A^F) C H^{X,Rom{F* , F)). □ 

Step 2. Rank 2 case: It suffices to consider the boundary case when degi^ = g — 1. Let 
-F be a general bundle in U{2,g — 1). Since a'^F = detF is general in Pic^~^(X), we have 
H^{X, A^F) = 0. Thus it suffices to show the vanishing of F°(X, Sym^F). 

First we show that there is no nonzero map F* ^ F whose image is of rank 1: Since 
si{F) = g — 1, every line subbundle of F has degree < 0. Furthermore, by [14^ Corollary 
3.2]), F has only finitely many maximal line subbundles (of degree 0). Thus if there were a 
nonzero map a: F* ^ F with image M of rank 1, then deg M = and F should have both 
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M and M* as its maximal subbundles. This implies either M = M* or there is a sequence 

for some torsion sheaf r of degree g — 1. By dimension counting, one can check that neither 
of these conditions is satisfied by a general F £ ly({2, g — I). 

Next we show that if Sym^F has a nonzero section a, then it does not extend to every 
deformation of F. We have seen that a : F* ^ F must be generically surjective. Hence 
the induced cohomology map 

a": H\X,End{F)) ^ H^{X,Rom{F* , F)) 

is surjective. By the assumptions {X , Sym^ F) > and x(Sym^F) = 0, we see that 
h^{X, Sym^F) > 0. This implies that the image of a" is not contained in H^{X, A^-F). By 
Lemma [531 does not extend to a for some deformation F of F. Therefore, H^iX, F(g)F) 
vanishes for a general F G U{2, g — 1). 

Step 3. Induction for higher rank cases: Now we consider bundles of rank n > 3. Firstly, 
suppose n is even. By semicontinuity, it suffices to find a bundle Fq € U{n, ^n{g — 1)) such 
that H^{X,Fo (g) Fo) = 0. We let Fq = Fi F2 where Fi is general in U{2,g - 1) and F2 
is general in U{n — 2, ^{n — 2){g — 1)), so that Fq is a polystable bundle of rank n and 
degree ^n((7 — 1). From the generality condition and the induction hypothesis, Fi (8>Fi and 
F2 ® F2 have no nonzero sections. Also hP{X, Fi (8) F2) = by Hirschowitz' lemma O 4.6], 
[19[ Theorem 1.2]. Therefore, Fq (8) Fq has no nonzero sections. 

Next suppose both n and g are odd. Let Fq = Fi © F2 for a general line bundle Fi of 
degree and a general F2 C ^/(n — 1, ^(n — l){g — 1)). By the same argument as above, 
we see that /i°(X,Fo (8F0) = 0. 

Finally suppose that n is odd and g is even. In this case, it suffices to find a bundle 
Fq G U{n, \ {n{g — 1) — 1)) such that H^{X, Fq ® Fq) = 0. Let Fq be a general extension 

^ Fi ^ Fo ^ F2 ^ 

for a general line bundle Fi of degree and a general F2 £ ly({n — 1, ^{n — l){g — 1)). 
From the stability of Fi and F2, it is easy to check that Fq is also stable. Again by the 
same argument as above, we see that h^{X, Fq (8> Fq) =0. □ 

Corollary 5.5. If F is a general bundle of rank n and degree d > ^n{g — 1), then 

h^{X, Sym^F) = (n + l)d - ^n{n + l){g- 1). 

Proof. For a theta characteristic k of X, we have 

h^{X,Sym'^F) ^ «> Sym^F*) 

= /i°(X,Sym2(K(8F*)). 
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Since deg{K(>Si F*) = n{g — l) — d < ^n{g — l), we can apply Lemma [5. 3 1 to get the vanishing 
of H^{X,Sym^F). By Riemann-Roch, we get the wanted result on h^^XjSym^F). □ 

Remark 5.6. In the case when either n is even or g is odd, the assignment F F ® F 
defines a morphism 

U{n,^n{g-l))^U{n\n\g-l)), 

since a tensor product of semistable bundles is semistable. The target moduli space has 
a generalized theta divisor, whose support consists of semistable bundles with nonzero 
sections. Lemma 15.31 implies that the image of this morphism is not contained in the 
generalized theta divisor. 
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